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Abstract 

A human-in-the-loop (HITL) simulation 
experiment was conducted by the National 
Aeronautics and Space Administration (NASA) 
to assess airline transport pilots ’ performance 
and reported acceptance of the use of 
procedures relying on airborne separation 
assistance and trajectory management tools. 
This study was part of a larger effort involving 
two NASA centers that includes multiple HITL 
experiments planned over the next few years to 
evaluate the use of automated separation 
assurance (SA) tools by both air traffic 
controllers and pilots. 

This paper presents results of measured 
pilot response delay that subject pilots incurred 
when interacting with cockpit tools for SA and 
discusses possible implications for future 
concept and procedures design. 

1 Introduction 

The automation of the separation function 
is expected to reduce the capacity limiting 
effect of human-based separation control of 
today’s system. Automation can remove 
workload bottlenecks, improve prediction of 
conflicts as traffic patterns become more dense 
and complex, and provide conflict-free 
maneuver alternatives to human operators. 
Future concepts of operations that are being 
evaluated as part of the Next Generation Air 
Transportation System (NextGen) [1] include 
both ground and aircraft based automation for 
separation. For example, integrated air/ground 
operational concepts have been proposed in 


which some aircraft crews exercise separation 
functions aided by Airborne Separation 
Assistance System (ASAS) tools on the flight 
deck, while air traffic controllers exercise 
ground based separation control for non- 
ASAS-equipped aircraft and terminal 
operations [2,3], ASAS systems integrate 
advanced decision-aiding automation into 
aircraft avionics that rely on broadcast 
surveillance information including aircraft 
velocity vectors and limited flight plan 
information through a surveillance capability 
such as the Automatic Dependent Surveillance 
- Broadcast (ADS-B). 

Automation systems for separation require 
operators to respond in a timely manner to 
conflict alerts in accordance with procedures 
and training. 

Delayed responses to conflict alerts may 
result in late resolutions possibly leading to 
loss of separation (LOS) as shown in Figure 1. 



Fig. 1. Delayed Traffic Conflict Resolutions 


There is very limited experience with 
human performance metrics when interacting 
with airborne separation automation since to 
date there are no operational systems and very 
few experimental studies. These applications 
are known to be context sensitive, in other 
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words, human performance is influenced by 
specific training and procedures as well as 
other required cockpit tasks. Multiple factors 
can affect pilot response time and pilot 
conformance, such as lack of trust as a result of 
frequent occurrences of false alerts or 
conflicting training instructions. Also, lack of 
situation awareness and excessive workload are 
factors known to affect response delay. The 
operational procedure within which the alerting 
system is used is likely to affect the operator’s 
response time [4], 

When pilots are given responsibility for 
maintaining separation and are trained and 
expected to react within a certain time period, 
then a response delayed beyond the specified 
time range can be considered a pilot error. The 
Traffic Alert and Collision Avoidance System 
(TCAS) II procedures specify that pilots are 
expected to act within 5 seconds of a 
Resolution Advisory. It is not expected that the 
procedures for separation provision will require 
as short a response time as collision avoidance. 
In research systems, the design of separation 
tools and procedures generally assume a pilot 
response time of several tens of seconds. 

Pilots operating under Visual Flight Rules 
(VFR), while not responsible for “separation” 
in the air traffic control sense, are expected to 
follow the established rules to see and avoid 
obstacles and other aircraft. While there are no 
expected response times specified in the 
procedures, the measured pilot response times 
are reported to be between 10 and 12.5 seconds 
according to the Federal Aviation 
Administration (FAA) and Department of 
Defense (DoD) [5], 

A recent batch 7 (non-piloted) experiment 
addressed the potential impact of aircraft crew 
response delay when interacting with ASAS 
tools using response delays between 5 and 240 
seconds in high density traffic [6]. This “stress 
test” experiment revealed great resilience of the 
strategic-only (i.e., flight-plan based) ASAS 
automation functions being tested, showing 
performance degradation only at combinations 
of extreme pilot delays and traffic density 


1 In batch experiments aircraft simulators are flown by 
an automated pilot model. 


levels. These results begin to shed light on 
future ASAS procedures and reliability 
requirements of strategic ASAS capability. The 
next step is to obtain measurements of actual 
response delays from HITL simulation 
experiments and flight tests and to assess the 
impact of observed delays on system 
performance. In addition, these studies will 
help to refine pilot procedures, flight deck 
interfaces and training guidelines. 

The HITL simulation experiment reported 
in this paper measured pilots’ response delay 
when interacting with ASAS automation under 
high traffic density conditions. Participants 
were current commercial transport pilots, with 
no prior experience with self separation 
concepts. They were given training on cockpit 
procedures and were expected to respond to 
traffic conflict alerts without delay. The 
independent variables of the study were traffic 
density and arrival time constraints. 

The remainder of the paper is organized as 
follows: Sections 2 and 3 provide a brief 
description of the simulation platform and 
experiment respectively. A more in-depth 
description of this experiment is provided in 
[7]. Section 4 describes the measures collected 
and the analysis of results and Section 5 the 
conclusion. 

2. Simulation Platform 

The simulation platform utilized in this 
experiment was the Airspace & Traffic 
Operations Simulation (ATOS) [8] that runs in 
the Air Traffic Operations Laboratory (ATOL) 
at the NASA Langley Research Center. It is 
comprised of hundreds of real-time, aircraft 
simulators equipped with ADS-B and self- 
separation capability. The HITL aircraft 
simulator is the Aircraft Simulation for Traffic 
Operations Research (ASTOR). Each ASTOR 
includes a six degree-of-freedom dynamic 
airplane model, a Flight Management System 
(FMS), Mode-S ADS-B datalink capability, 
and a prototype ASAS called the Autonomous 
Operations Planner (AOP) [9]. 

The AOP provides an integrated suite of 
capabilities for managing traffic conflicts and 
trajectory changes from the flight deck 
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perspective, including conflict detection, 
resolution, prevention, and trajectory constraint 
conformance. These capabilities are developed 
to a level of fidelity sufficient for research of 
complete airborne responsibility for self- 
separation as defined by the Principles of 
Operation of ASAS Category 4, “Airborne 
Self-Separation” [10], In the Joint Planning and 
Development Office (JPDO) NextGen Concept 
of Operations, they correspond to the functions 
of Separation Management and aspects of 
Trajectory Management [1], 

2.1 The AOP Pilot Interface 

The AOP conflict detection and resolution 
logic includes a strategic intent-based function 
and a tactical intent-based function. (The 
tactical state-based function was not active in 
this experiment.) The strategic system is used 
for FMS-based operations. The tactical system 
is used for flight using Mode Control Panel 
(MCP) guidance and serves primarily as a 
“tactical override” of the strategic system when 
predicted time to LOS is short. Conflict 
alerting is modeled as a multi-alert-level 
system in which the timing of alerts between 
aircraft are staggered as a method for 
incorporating a right-of-way rule set (i.e., 
priority rules), based roughly on the set used in 
VFR. The AOP provides pilots with two 
different conflict alert levels depending on the 
predicted time to LOS. A Level 1 (LI) conflict 
alert is displayed 6-10 minutes (min) before a 
predicted LOS. A Level 2 (L2) conflict alert is 
displayed 0-6 min before a predicted LOS. 

Both alerts are displayed on the 
Navigation Display (ND) with an overlaid 
white (LI) and yellow (L2) “dog bone” shape 
on the ownship’s active route as shown in 
Figure 2. The “dog bone” length indicates the 
location and duration of the predicted LOS. LI 
alerts are accompanied by a chime sound and 
L2 alerts are also accompanied by an aural 
warning saying “traffic alert traffic alert”. 
Furthermore, L2 alerts come with a countdown 
timer, whereas LI alerts do not. The 
countdown timer indicates how much time the 
pilots have to execute a strategic resolution 


before “tactical override” compels them to 
solve the conflict tactically. 

Pilots are trained and expected to respond 
to these types of alerts in a timely manner, 
according to the severity of the conflict alert. 



Fig. 2: AOP Interface: Traffic Conflict 
Alert 


Most conflicts are LI alerts when first 
detected and may progress to an L2 alert if a 
pilot fails to respond on time. On rare 
occasions, usually due to blunders or severe 
deviations from the predicted flight path, a 
conflict may be detected late and displayed as 
an L2 alert from the beginning. Pilots are 
trained to perform the same actions for L 1 and 
L2 alerts (i.e., choose and execute a resolution 
trajectory) but with an increased sense of 
urgency in the latter case. The conflict 
detection and resolution (CD&R) timeline is 
described in the next Section. 

2.2 The Conflict Detection and Resolution 
Timeline 

As shown in Figure 3, the timeline for the 
life span of a strategically resolved conflict by 
the AOP, a conflict begins at first detection, but 
for the pilot, it begins when it is first displayed 
on the ND. The earliest display time for 
aircraft with higher priority is 7 min to a 
predicted LOS. The “give-way” aircraft gets 
the alert as soon as the conflict is detected. 
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The pilot response delay d i, or “realization 
time,” is the time elapsed from the first display 
of the conflict alert to the pilot’s reaction of 
requesting a resolution from the AOP. A 
distracted or busy pilot that fails to notice a 
conflict may exhibit longer di delay. Next, the 
resolution cycle time is the time it takes AOP 
to compute one or more conflict-free trajectory 
alternatives. This time is variable and can be 
affected by traffic density and complexity [11], 
The cycle ends when the resolution trajectory is 
displayed to the pilot. The time elapsed from 
resolution display to the pilot’s action of 
uploading a resolution route to the FMS is the 
response delay d 2 . At this point, pilots are 
already aware of the conflict and are 
considering the resolution route offered by 
AOP. During this “selection time” the pilot 


carefully evaluates the proposed vertical and or 
lateral resolution routes by examining both on 
the ND. Once the resolution route is uploaded 
to the FMS, it is displayed as a white dashed 
line and needs to be executed to become the 
new active route. The pilot response delay d 3 is 
the time elapsed from the pilot’s upload action 
to the action of executing the new FMS route. 
This “decision time” gives the pilot the option 
to reconsider the new route before committing 
to the change. 

It must be noted that the chart in Figure 3 
represents a nominal timeline in which there 
are no intervening events. Frequently, other 
events such as new traffic conflicts can occur 
during any of the phases described. Those cases 
need to be treated individually to identify the 
timing of the pilot actions. 
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Fig. 3. Conflict Detection and Resolution Timeline 


3 Experiment Description 

The study reported in this paper was 
part of a larger effort involving NASA Ames 
and Langley Research Centers. It comprises 
multiple HITL experiments planned over the 
next few years to evaluate the use of automated 
separation assurance (SA) tools by both air 
traffic controllers and pilots. 

The primary goal of this first 
experiment was to assess the degree of 
comparability achievable with a companion 
HITL experiment conducted to investigate 
ground-based automated separation [12], 


An additional goal of the experiment 
was to assess the agility of self-separation 
operations in managing trajectory-changing 
events in high traffic density, en-route 
operations with arrival time constraints. 
Secondary goals included assessing pilot 
acceptability of the concept, tools, and 
procedures; collecting pilots’ subjective 
workload ratings; and objectively measuring 
safety, efficiency, and pilot performance. This 
paper presents results of measured pilot 
response delay that subject pilots incurred 
when interacting with cockpit tools for SA. 
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3.1 Experiment Design 

The experiment included two sets of 
scenarios: one set of four 30-min scenarios, and 
one set of three 15-min scenarios. 

A 2x2 within-subjects design was used to 
collect data during the 30-min scenarios that 
involved either the presence or absence of 
scheduling assignments provided to aircraft 
operating within an airspace having a sustained 
traffic density level either 1.5 times (1.5x) or 2 
times (2. Ox) greater than approximate current 
day airspace capacity. Two replicates of each 
30-min scenario were performed for a total of 
eight simulation runs. 

A separate 3x1 within-subject design was 
used to collect data during three 15-min 
experiment scenarios. The three 15-min 
scenarios had a 2. Ox traffic density level, and 
two of the 15-min scenarios included scripted 
events that manipulated the timing of aircraft 
trajectory changes. These changes were 
induced by a data link message containing an 
updated required time of arrival (RTA) 
necessitating a delay maneuver. Two replicates 
of each 15-min scenario were performed for a 
total of six simulation runs. 

3.2 Participants and Training Approach 

Data were analyzed from forty-eight 
commercial transport pilots. Two of these 
pilots had not flown a transport category 
aircraft within the last year. However, the 
remaining 46 pilot participants consisted of 38 
commercial transport pilots employed by U.S. 
air carriers or aircraft manufacturers and eight 
commercial transport pilots employed by 
European air carriers. 

The 48 subject pilots participated as 
groups of 12 in four separate three-day 
experiment sessions conducted in March 2010. 
Each experiment session consisted of a series 
of training exercises involving classroom 
instruction and hands-on simulated flight, the 
completion of 14 data collection scenarios, and 
a final debrief session involving the pilot 
participants and members of the research team. 

During their training exercises, pilots 
received instruction regarding the self- 


separation operational concept and associated 
cockpit technology and procedures. Timeliness 
of responses to traffic alerts was emphasized, 
and pilots were instructed that, when notified of 
a traffic conflict, they were expected to resolve 
the conflict without delay and to execute one of 
the offered resolutions in a timely fashion [7]. 
However, the pilots were not given a specific 
response time requirement (e.g., 5 seconds for 
TCAS). It was expected that the pilots would 
take the time necessary following the alert to 
understand the conflict situation before acting. 

4. Results and Discussion 

Pilot response times recorded during the 
30-min and 15-min scenarios were analyzed 
separately to determine if the duration and 
other substantial scenario differences had an 
effect on response time. Differences included 
the number of observed conflicts (more 
conflicts were observed during the 30-min 
scenarios), proximity of the top-of-descent 
(often reached in the 30-min scenarios but 
never in the 15-min scenarios), and inclusion of 
the scripted data link required time of arrival 
(RTA) message in the 30-min scenarios. 

Data from the 30-min scenarios are 
presented first. There were a total of 242 
strategically resolved 2 conflicts during the 30- 
min scenarios. Of these conflicts, 230 were 
encountered by all 48 pilots and first appeared 
as LI alerts. The remaining 12 conflicts first 
appeared as “pop-up” L2 alerts and were 
encountered by 11 different pilots. For each 
pilot, the mean values of di, d 2 , and d 3 were 
computed over all eight of the experiment’s 30- 
min scenario runs. 

Response times to LI and L2 conflicts 
were analyzed separately to assess the effect of 
alert level and its associated urgency on pilot 
response time. 

4.1 LI Alerts for the 30-min Scenarios 

4.1.1 Pilot Response Delays d / 

A histogram of di for the 48 pilots is 
shown in Figure 4. The data are right skewed, 


2 Conflicts resolved tactically (using the MCP rather than 
the FMS) were excluded from this initial analysis. 
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and there is an outlier at 84.20 seconds (sec). 
The mean pilot response delay di for the 48 
pilots is 15.37 sec, and the standard deviation is 
11.58 sec, with a minimum of 5.38 and a 
maximum of 84.20 sec. 

From the histogram in Figure 4, the 
pilot response delay di for LI conflict alerts 
appears to approximately follow a lognormal 
distribution with estimated parameters ft = 
2.60 and a = 0.46. 



Fig. 4. Histogram of di with lognormal 
distribution fit (30-min scenarios) 

A 95% confidence interval on the mean 
pilot response delay di is (11.77, 15.38). In 
other words, we are 95% confident that the true 
mean lies between 11.77 and 15.38 sec. 

4.1.2 Pilot Response Delays d? 

A histogram of the pilot response delay 
d 2 for the 48 pilots is shown in Figure 5, and 
we can see that the data are right-skewed. 



Fig. 5. Histogram of d 2 with lognormal 
distribution fit (30-min scenarios) 

The mean pilot response delay d 3 for 
the 48 pilots is 9.91 sec and the standard 
deviation is 3.94 sec, with a minimum of 5.35 


and a maximum of 25.50 sec. From the 
histogram in Figure 5 the pilot response delay 
do appears to approximately follow a lognormal 
distribution with estimated parameters ft = 
2.23 and a = 0.35. A 95% confidence 
interval on the mean pilot response delay d 2 is 
(8.41,7.51). 

4.1.3 Pilot Response Delays di 

A histogram of the pilot response delay 
d 3 for the 48 pilots is shown in Figure 6, and 
once again the data are right skewed. The mean 
pilot response delay d 3 for the 48 pilots is 7.24 
sec and the standard deviation is 3.17 sec, with 
a minimum of 1.85 and a maximum of 16.37 
sec. From the histogram in Figure 6, the pilot 



Fig. 6. Histogram of d 3 with lognormal 
distribution fit (30-min scenarios) 

response delay d 3 appears to approximately 
follow a lognormal distribution with estimated 
parameters ft = 1.89 and d = 0.44. A 95% 
confidence interval on the mean pilot response 
delay d 3 is (5.80, 7.51). 

A summary of the descriptive statistics 
for response delays di, d 3 , and d 3 to LI conflict 
alerts is shown in Table 1. 


Table 1: Descriptive statistics for pilot 
response delays di, d 2 , and d 3 for LI conflict 
alerts and 48 pilots (30-min scenarios) 



Mean 

StDev 

Min 

Med 

Max 

95% 

Cl 

d\ 

15.37 

11.58 

5.38 

12.53 

84.20 

(11.77, 

15.38) 

d-2 

9.91 

3.94 

5.35 

8.62 

25.50 

8.41, 

10.29 

d?, 

7.24 

3.17 

1.85 

6.45 

16.37 

(5.80, 

7.51) 
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All three response times seem to be 
similarly distributed. The mean response time 
to a conflict’s first detection is approximately 
16 sec and the mean time to evaluate the 
resolution routes, 62 is about 10 sec. This is the 
time during which pilots were evaluating and 
comparing the resolution routes offered by 
AOP (vertical and lateral maneuvers) before 
uploading a particular route. 


Table 2. Lognormal distribution estimated 
parameter values for pilot response delays 
dl, d2, and d3 for LI alerts (30-min 
Scenarios) 


Variable 


O 

d\ 

2.60 

0.46 

d 2 

2.23 

0.35 


1.89 

0.44 


After uploading the route it took pilots 
approximately 8 sec to execute the route. Many 
factors are likely to have affected those times; 
among them new procedures and 
responsibilities, training, trust in the system, 
interface design, experiment modeling 
limitations, etc. 



Fig. 7. Lognormal distribution fit of di, d 2 , 
and d 3 for LI alerts (30-min scenarios) 

At the end of the experiment, pilot 
participants provided abundant feedback to that 
effect which will be assimilated in new design 
guidelines for the AOP interface and pilot 


procedures. In future batch 3 simulation 
experiments, these pilot response delays can be 
simulated as constant values by using the 
means in Table 1. They can also be randomly 
generated using the lognormal distribution with 
the estimated parameters in Table 2, which 
would have the probability density functions in 
Figure 7. 

4.2 L2 Alerts for the 30-min Scenarios 

Of the 242 strategically resolved 
conflicts, only 12 were first displayed as L2 
alerts. These unusual “pop-up” conflicts were 
encountered by 11 different pilots. The mean 
values of di, d^, and ds were computed for each 
pilot and then aggregated over all the pilots. It 
should be noted that this analysis is based on a 
small dataset. Descriptive statistics are 
summarized in Table 3. 


Table 3: Descriptive statistics for pilot 
response delays di, d 2 , and d 3 for L2 conflict 
alerts and 11 pilots (30-min scenarios) 



Mean 

StDev 

Min 

Med 

Max 

95% 

Cl 

d\ 

7.65 

5.49 

3.15 

6.80 

23.00 

(4.46, 

9.48) 

^2 

7.87 

3.86 

4.25 

6.60 

17.55 

(5.48, 

9.52) 

di 

5.10 

3.56 

1.50 

4.60 

14.30 

(2.78, 

6.46) 


As shown in Tables 1 and 3, the 
response delays for the “pop-up” L2 alerts are 
much smaller than for the LI alerts, particularly 
di. This finding is consistent with the expected 
higher sense of urgency intended by these 
conflict alerts. The mean initial reaction time di 
for L2 alerts was approximately half of the 
corresponding time for LI alerts. 

There are multiple reasons to explain 
this behavior but clearly the interface design 
for LI and L2 alert levels combined with the 
yellow symbology, appear to have had the 
intended effect of shortening the pilot response 
time. It is also possible that pilots waited longer 
to act on LI alerts to see if the conflict was 


3 In batch experiments aircraft simulators are flown by 
an automated pilot model. 
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resolved by the other aircraft or was a false 
alert. 

4.3 LI and L2 Alerts for the 15-min 
Scenarios 

There were a total of 57 strategically 
resolved conflicts during the experiment’s 15- 
min scenarios. Of these conflicts, 56 were LI 
conflicts, and only one was a “pop-up” L2 
conflict. Conflicts were encountered during the 
15-min scenarios by only 29 of the 48 pilots. 
For each pilot, the mean values of di, d 2 , and d 3 
were computed over all strategically resolved 
conflicts that he encountered during his six 
runs. 


Table 4: Descriptive statistics for pilot 
response delays di, d 2 , and d 3 for all conflict 
alerts and 29 pilots (15-min Scenarios) 



Mean 

StDev 

Min 

Med 

Max 

95% 

Cl 

d\ 

14.15 

7.49 

6.70 

11.35 

35.45 

(10.75, 

15.06) 

d.2 

10.81 

5.53 

3.60 

9.30 

26.90 

(8.20, 

11.56) 

di 

7.26 

5.63 

2.40 

5.15 

27.95 

(4.58, 

7.46) 


Table 4 presents a summary of the 
descriptive statistics for pilot response delays 
di, d 2 , and d 3 , and confidence intervals on the 
mean delays. A comparison of these data with 
those shown in Tables 1 and 3 indicates that for 
the 29 pilots that encountered strategic 
conflicts, the mean pilot response delays 
recorded during the 15-min scenarios are 
similar to those recorded during the 30-min 
scenarios for the 29 pilots that encountered 
strategic conflicts. 


Table 5. Lognormal distribution estimated 
parameter values for pilot response delays 
di, d 2 , and d 3 (15-min scenarios) 


Variable 


d 

d\ 

2.54 

0.44 

^2 

2.28 

0.45 

d% 

1.77 

0.63 


In fact, 29 of the 48 pilots encountered 
conflicts in both the 15-min and 30-min 
scenarios. For these pilots, the scenario 
differences had no significant effect on mean 
pilot response delay di, d 2 , or d 3 using alpha = 
0.05. A total of five outliers were encountered 
ranging between 27.95 and 36.90 seconds. 



0 10 20 30 40 


Pilot Response Delay (sec) 

Fig. 8. Lognormal distribution fit of di, d 2 
and d 3 for L2 alerts (15-min scenarios) 

In future non-piloted simulation 
experiments, these pilot response delays can be 
simulated as constant values by using the 
means in Table 4. They can also be randomly 
simulated using the lognormal distribution as 
shown in Figure 8 with the estimated 
parameters in Table 5. 

5 Conclusion 

Separation assurance automation must be 
designed to account for, and mitigate the 
impact of varying human response. This must 
be accomplished through extensive testing and 
evaluation of new technologies involving F1ITL 
experimentation and batch “stress” tests. The 
former allows the collection of measurements 
of human operator performance variability 
while the latter is needed to address the 
resilience of the new systems using 
performance models based on the experimental 
measures. This process will enable the design 
of preventive procedures and more robust 
automation. 

This paper provided an in-depth 
analysis of the pilot response delay measures 
obtained during a recent HITL experiment that 
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involved forty eight commercial transport 
pilots. 

A lognormal distribution was found to 
be a good fit for the pilot response delays and 
appropriate coefficients for the individual delay 
components di, d 2 , and d 3 were provided. These 
results can be applied in future experiments to 
implement pilot performance models for 
similar applications. Response delay times can 
be modeled using the values provided in Tables 
1 and 3 or randomly generated using the 
lognormal distribution with the coefficients 
provided in Tables 2 and 5. 

Additionally, it was determined that the 
total pilot response times were well within 
ranges verified during prior stress tests and no 
safety impact in terms of LOS was observed. 

The interface design methods intended 
to reduce pilot response in more urgent 
situations (L2 alerts) had the desired effect of 
getting the attention of the pilots and reducing 
the overall CD&R time. 

Finally, results from this study will 
contribute to the understanding of aircraft 
crews’ subjective perception of their new role 
and responsibility in a self separation concept 
of operations and provide design guidelines for 
new procedures and tools. 
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